Skip to content

test(e2e): add semantic live progress phases#7100

Merged
cv merged 27 commits into
mainfrom
codex/diagnose-agent-turn-latency-hang
Jul 22, 2026
Merged

test(e2e): add semantic live progress phases#7100
cv merged 27 commits into
mainfrom
codex/diagnose-agent-turn-latency-hang

Conversation

@cjagwani

@cjagwani cjagwani commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Live E2E jobs now report the semantic phase they are executing without repeating Vitest's test-level start, identity, elapsed time, or completion output. Every live test declares an individualized ordered plan, and each completed phase records its outcome and duration so a stalled or failed matrix job shows where time was spent.

Related Issue

Part of #7101

Changes

  • Instrument all 121 collected live E2E cases across 80 live entries (79 test implementations plus one bootstrap workflow forwarder) with ordered, test-specific semantic phases.
  • Extend the shared E2E fixture to print low-noise phase transitions with passed, failed, or skipped outcomes and per-phase durations, then append a harness-owned resource-release phase so cleanup hangs and failures are timed independently.
  • Preserve the originating phase for hard failures, soft assertions, and skips; persist the complete timeline, target, and shard identity in test-progress.json, including the dedicated skill-agent job upload.
  • Emit secret-safe stall evidence after five minutes and every ten minutes thereafter, including phase/output age, bounded command activity, process memory, free memory, disk, load, and the existing runner-pressure snapshot.
  • Add a collection-only checker and pre-commit gate that require every live test to use the shared fixture, declare a valid semantic plan, enter every later phase, and keep transitions scoped to their owning test, including siblings and repeated registrations that share one plan. The checker redirects the one compiled-artifact live import to source only while collecting, so it works in a clean checkout without running live bodies; fixture-backed negative coverage verifies missing metadata plus nonliteral, undeclared, and sibling-owned transitions are rejected.
  • Keep the historical runtime audit and matrix progress-summary upload from the earlier PR commits so maintainers can rank slow or unreliable targets and inspect phase artifacts across shards.
  • Document phase authoring, cleanup ownership, outcome/duration semantics, stall output, artifacts, and runtime-audit usage in the contributor and E2E guides.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 4 focused final-head E2E-support files, 28 tests passed; semantic collection audit passed for 121 tests across 80 live entries; mock/live parity, conditional-growth, npm run typecheck:cli, project membership, source-shape, file-size, changed-file Biome, and git diff --check passed
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: required CI is authoritative for the broad sharded suite
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — validation passed; Fern reported 2 existing warnings
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Summary by CodeRabbit

  • New Features

    • Live E2E runs now emit ordered phase markers with clear outcomes and durations, plus periodic stall diagnostics.
    • Each live E2E run produces a test-progress.json artifact (including target/shard metadata when available).
    • Added semantic phase-plan validation to prevent missing/invalid phase coverage before running.
    • Added runtime auditing to rank captured runs by median/p95/variability and identify the slowest phase.
  • Documentation

    • Expanded E2E fixture and workflow guidance for phase reporting, diagnostics, and runtime analysis.
  • Tests

    • Added automated checks for phase validity, progress outcomes, runtime auditing, and artifact upload requirements.

@cjagwani cjagwani self-assigned this Jul 17, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds shared E2E phase tracking, shell instrumentation, persisted test-progress.json summaries, runtime aggregation, semantic phase validation, CI artifact publication, and migration of live tests to declared phase plans.

Changes

E2E runtime observability

Layer / File(s) Summary
Shared progress tracking
test/e2e/fixtures/progress.ts, test/e2e/fixtures/e2e-test.ts, test/e2e/support/*progress*
Adds validated phase plans, phase timing, stall diagnostics, resource evidence, output tracking, teardown outcomes, and persisted progress summaries.
Shell activity and live-test integration
test/e2e/fixtures/shell-probe.ts, test/e2e/live/*, test/e2e/support/*
Threads progress reporting through shell commands, retries, cleanup, inference flows, and the live E2E suite.
Semantic phase validation
tools/e2e/check-semantic-phases.mts, vitest.config.ts, .pre-commit-config.yaml, package.json
Adds static collection and validation of phase metadata, phase calls, fixture imports, and scoped test coverage.
Runtime audit and artifact publication
scripts/audit-test-runtime.mts, .github/workflows/e2e.yaml, tools/e2e/*, CONTRIBUTING.md, test/e2e/README.md, test/e2e/docs/README.md
Adds progress artifact discovery, validation, percentile aggregation, Markdown output, CLI scripts, workflow uploads, boundary checks, documentation, and audit tests.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LiveE2ETest
  participant TestProgress
  participant ShellProbe
  participant E2EArtifact
  participant RuntimeAudit
  LiveE2ETest->>TestProgress: declare and advance semantic phases
  LiveE2ETest->>ShellProbe: execute instrumented command
  ShellProbe-->>TestProgress: report output and activity
  TestProgress->>E2EArtifact: write test-progress.json
  RuntimeAudit->>E2EArtifact: discover and validate summaries
  RuntimeAudit-->>RuntimeAudit: rank p95, variability, max, and slowest phase
Loading

Possibly related issues

Possibly related PRs

  • NVIDIA/NemoClaw#6672: Both changes integrate capabilities through the shared E2E fixture and Hermes live test.
  • NVIDIA/NemoClaw#6969: Both coordinate CI shard identity with live E2E artifact and progress metadata.

Suggested labels: chore

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 2.17% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately summarizes the main change: adding semantic live progress phases to E2E tests.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/diagnose-agent-turn-latency-hang

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 6432d2e in the codex/diagnose-agent... branch remains at 96%, unchanged from commit f992494 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 6432d2e in the codex/diagnose-agent... branch remains at 80%, unchanged from commit f992494 in the main branch.

Show a code coverage summary of the most impacted files.
File main f992494 codex/diagnose-agent... 6432d2e +/-
src/lib/onboard...ndbox-create.ts 83% 33% -50%
src/lib/onboard...-create-plan.ts 88% 75% -13%
src/lib/onboard...-desktop-gpu.ts 89% 77% -12%
src/lib/onboard...ndbox-create.ts 91% 83% -8%
src/lib/domain/.../connect-env.ts 97% 89% -8%
src/lib/messagi...nnels/policy.ts 100% 100% 0%
src/lib/sandbox...rce-identity.ts 87% 87% 0%
src/lib/securit...ntial-filter.ts 93% 93% 0%
src/lib/shields/index.ts 71% 72% +1%
src/lib/platform.ts 84% 89% +5%

Updated July 22, 2026 14:57 UTC

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, agent-turn-latency, bedrock-runtime-compatible-anthropic, brave-search, channels-add-remove, channels-stop-start, cloud-inference, common-egress-agent, concurrent-gateway-ports, credential-migration, cron-preflight-inference-local, device-auth-health, diagnostics, docs-validation, double-onboard, full-e2e, gateway-guard-recovery, gateway-health-honest (+53 more)

1 optional E2E recommendation
  • e2e-all

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure feature PR adds or expands user-visible functionality labels Jul 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@cjagwani cjagwani changed the title test(e2e): expose agent turn latency progress test(e2e): add shared progress diagnostics and runtime audit Jul 20, 2026
@cjagwani
cjagwani marked this pull request as ready for review July 20, 2026 14:53
@cjagwani
cjagwani marked this pull request as draft July 20, 2026 15:01
@cjagwani
cjagwani marked this pull request as ready for review July 20, 2026 15:02
@cjagwani
cjagwani marked this pull request as draft July 20, 2026 15:03

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/audit-test-runtime.mts`:
- Around line 99-100: Update scripts/audit-test-runtime.mts at lines 99-100 to
use a dedicated median helper that averages the two middle sorted durations for
even-sized run sets, while preserving the existing percentile calculation for
p95. Update test/e2e/support/test-runtime-audit.test.ts at lines 56-82 to expect
the two-run median, variability, and formatted row as 30.0s, 20.0s, and 20.0s.

In `@test/e2e/live/agent-turn-latency.test.ts`:
- Around line 50-93: Remove the explicit progress.stop() call associated with
the cleanup setup around cleanup.trackDisposable. Let cleanup.runAll() execute
while progress remains active so cleanup phases and output are preserved,
relying on the progress fixture’s finally block to stop itself.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9573d819-5d22-4bb1-a36a-ae48ae47dc38

📥 Commits

Reviewing files that changed from the base of the PR and between 949b5f9 and e743391.

📒 Files selected for processing (17)
  • CONTRIBUTING.md
  • package.json
  • scripts/audit-test-runtime.mts
  • test/e2e/README.md
  • test/e2e/docs/README.md
  • test/e2e/fixtures/e2e-test.ts
  • test/e2e/fixtures/progress.ts
  • test/e2e/fixtures/shell-probe.ts
  • test/e2e/live/agent-turn-latency-helpers.ts
  • test/e2e/live/agent-turn-latency.test.ts
  • test/e2e/live/rebuild-hermes-progress.ts
  • test/e2e/live/rebuild-hermes.test.ts
  • test/e2e/support/agent-turn-latency-progress.test.ts
  • test/e2e/support/e2e-fixture-context.test.ts
  • test/e2e/support/rebuild-hermes-progress.test.ts
  • test/e2e/support/rebuild-hermes-timing.test.ts
  • test/e2e/support/test-runtime-audit.test.ts
💤 Files with no reviewable changes (1)
  • test/e2e/live/rebuild-hermes-progress.ts

Comment thread scripts/audit-test-runtime.mts Outdated
Comment thread test/e2e/live/agent-turn-latency.test.ts
@cjagwani
cjagwani marked this pull request as ready for review July 20, 2026 15:10
@cjagwani

Copy link
Copy Markdown
Collaborator Author

Fresh advisor follow-up: PRA-1 is addressed in aecce15 with a fixture-backed teardown test that verifies a completed version-1 summary, scenario, target ID, shard ID, duration, and final phase. PRA-2 does not reproduce: the live matrix sets E2E_ARTIFACT_DIR to e2e-artifacts/live (not the target directory), liveTargetTestName(target) is exactly target.id, and createArtifactSink joins those into e2e-artifacts/live/${matrix.id}/test-progress.json. The existing artifact-path test also asserts that exact root. Changing the upload contract to */test-progress.json would therefore miss the registry target artifact.

@cjagwani

Copy link
Copy Markdown
Collaborator Author

The growth-guardrail failure is addressed in 950fd9c: the fixture regression now uses Vitest environment stubs and contains no added if statements. The focused fixture test and the repository test-conditionals scan both pass.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/e2e-progress-fixture.test.ts`:
- Around line 34-39: Consolidate the environment restoration logic in the test
fixture cleanup into a data-driven loop over the saved variable names and
values, using a single conditional to delete variables whose previous value is
undefined or restore defined values. Update the restoration block containing
previousArtifactDir, previousTargetId, and previousShardId while preserving each
variable’s existing mapping and behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b64a9b47-60f5-4b69-9f15-2584904f691d

📥 Commits

Reviewing files that changed from the base of the PR and between f293e65 and aecce15.

📒 Files selected for processing (1)
  • test/e2e/support/e2e-progress-fixture.test.ts

Comment thread test/e2e/support/e2e-progress-fixture.test.ts Outdated
cv added 3 commits July 22, 2026 06:52
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv
cv merged commit fe7ab66 into main Jul 22, 2026
52 of 55 checks passed
@cv
cv deleted the codex/diagnose-agent-turn-latency-hang branch July 22, 2026 15:10
@cv cv mentioned this pull request Jul 22, 2026
22 tasks
cv added a commit that referenced this pull request Jul 22, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical `## v0.0.92` release entry to
`docs/changelog/2026-07-22.mdx` before the release plan is generated.
The entry summarizes all ten pull requests merged after v0.0.91,
including the OpenClaw security update and Jaeger runtime regression
coverage.

## Changes

- Added the canonical v0.0.92 changelog entry.
- Recorded the user-visible, security, documentation, CI, and test
changes in the release range.
- #7280 -> `docs/changelog/2026-07-22.mdx`: OpenClaw 2026.7.1 and
Node.js 22.23.1 security/runtime update.
- #7378 -> `docs/changelog/2026-07-22.mdx`: canonical macOS watcher path
validation.
- #7379 -> `docs/changelog/2026-07-22.mdx`: stabilized full WSL platform
validation.
- #7380 -> `docs/changelog/2026-07-22.mdx`: bounded swap for hosted
Hermes image exports.
- #7100 -> `docs/changelog/2026-07-22.mdx`: semantic progress phases for
live E2E tests.
- #7376 -> `docs/changelog/2026-07-22.mdx`: restored v0.0.91 changelog
history and corrected tagged guidance.
- #7374 -> `docs/changelog/2026-07-22.mdx`: reviewed Homebrew formula
transition for installer integrity checks.
- #7346 -> `docs/changelog/2026-07-22.mdx`: provider-neutral headless
server deployment guidance.
- #7381 -> `docs/changelog/2026-07-22.mdx`: stabilized Hermes guard
timing and WSL ownership fixtures.
- #7339 -> `docs/changelog/2026-07-22.mdx`: real-artifact Jaeger header
remediation regression coverage.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates the canonical dated changelog
and release heading contract.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts` passed 6/6 tests.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to a
changelog-only change.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — passed
with 0 errors and 2 pre-existing Fern warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added `v0.0.92` release notes covering sandboxing updates
(OpenClaw/Node.js bumps, integrity pinning remediation, mcporter
handling, and upgrade validation).
* Updated deployment guidance for provider-neutral headless installs,
and improved live E2E test reporting plus phase-plan validation.
* Tightened installer integrity-check messaging during an OpenShell
Homebrew transition and expanded platform/image validation (including
macOS/WSL timing) and hosted image export behavior.
* Restored the previously missed `v0.0.91` changelog entry and release
validation guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
cv added a commit that referenced this pull request Jul 22, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Final-main E2E previously fanned out even when the newest applicable
base-image publication was missing, partial, or failed. This change adds
a trusted, fail-closed preflight that proves all required publishers
succeeded before matrix generation, while preserving the
controller-selected PR path without checking out PR code.

## Related Issue

Fixes #7372

## Changes

- Add a dependency-free publication verifier that derives publisher
inputs from the checked-in workflow, resolves the applicable
first-parent commit, bounded-polls paginated GitHub evidence, validates
workflow/run/job provenance across rerun attempts, and reconfirms the
selected run.
- Add a least-privilege `base-image-publication` job before
`generate-matrix`; trusted main schedule/manual runs perform the check,
controller-selected PR dispatches bypass it before checkout, and invalid
invocation modes fail closed.
- Add focused behavioral and workflow-boundary coverage for pagination,
retries, batched pushes, merge history, partial/failed evidence,
diagnostic context, controller bypass, and trust-boundary mutations.
- Keep this change independent of #7100's progress artifacts and #7262's
matrix timeout; preserve those adjacent lines if either PR lands first.
- Scope this fix to publication sequencing. Strict digest pinning and
removal of downstream mutable `latest` fallback remain follow-up work.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: This changes internal GitHub
Actions sequencing and provenance validation only; CLI, configuration,
runtime, installation, and user-facing image behavior are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: A final fail-closed
trust-boundary review found one diagnostic-context gap, which was fixed
and regression-tested; no other blocking findings remained.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed
implementation
- Result: `no-docs-needed`
- Evidence: The change adds an automatic internal E2E
publication-provenance gate without changing CLI, configuration,
runtime, installation, or user-facing image behavior.
- Agent: Codex Desktop
- PR: #7373
<!-- docs-review-head-sha: 10a2123 -->
<!-- docs-review-agents-blob-sha: 73549ba -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run --project e2e-support
test/e2e/support/base-image-publication.test.ts
test/e2e/support/base-image-publication-workflow-boundary.test.ts
test/e2e/support/e2e-workflow.test.ts` — 95 passed.
- [x] Applicable broad gate passed — `npm run checks`, `npm run
typecheck:cli`, and `npx prek run --from-ref origin/main --to-ref HEAD`
passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a base-image publication verification gate that determines
whether publication evidence is required before running E2E workflows.
* Updated E2E matrix generation and PR E2E reporting to wait for this
gate’s result.
* Strengthened the E2E workflow boundary validation and approval
authorization checks to rely on stable, trusted attributes.
* **Tests**
* Added end-to-end boundary coverage for the base-image publication
policy, including classifier behavior and drift detection.
* Added evidence validation tests for workflow/run selection, publisher
job evidence rules, pagination safety, and deterministic polling/timeout
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
timothybrush pushed a commit to timothybrush/NemoClaw that referenced this pull request Jul 23, 2026
## Summary

Follow-up to NVIDIA#7100: scheduled and ordinary manual E2E runs now publish a
current-run semantic phase runtime table in the existing GitHub Actions
scorecard. The table reuses the artifact audit instead of adding
live-log noise, and the resume test no longer reports a bookkeeping-only
phase that consistently completed in zero seconds.

## Changes

- download this run's E2E artifacts in the read-only scorecard job and
append the existing runtime audit table to the GitHub step summary
- report the slowest observed phase's duration and `passed`, `failed`,
or `skipped` outcome
- keep missing or invalid progress evidence diagnostic-only so it cannot
hide the main E2E result
- remove `record the completed resume contract` from `onboard-resume`;
the final meaningful comparison phase now owns target completion
- protect the workflow boundary, native Node loading path, audit
rendering, and phase plan with regression tests
- document the automatic current-run table and the existing multi-run
audit command

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: no user-facing product
behavior changed; the E2E maintainer guides are updated.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: the scorecard remains
read-only (`actions: read`, `contents: read`), uses an immutable
artifact-action pin, treats download/audit failure as diagnostic-only,
and has workflow-boundary regression coverage.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed
implementation
- Result: `docs-updated`
- Evidence: `test/e2e/README.md` and `test/e2e/docs/README.md` document
the current-run GitHub Actions scorecard table and its phase-duration
and outcome fields.
- Agent: Codex Desktop
- PR: NVIDIA#7388
<!-- docs-review-head-sha: c190bea -->
<!-- docs-review-agents-blob-sha: 560ff38 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: not applicable
- Station profile/scenario: not applicable
- Result: not applicable
- Supporting evidence: not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npm exec vitest -- run --project
e2e-support test/e2e/support/test-runtime-audit.test.ts
test/e2e/support/e2e-scorecard.test.ts
test/e2e/support/e2e-scorecard-coordinator.test.ts
test/e2e/support/e2e-operations-workflow-boundary.test.ts` (61 passed);
`npm run test:e2e-phases:check` (121 tests across 80 files); `npm run
test:projects:check`; `npm run typecheck:cli`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: full `e2e-support` run
reached 1,343 passes; five discovery timeouts passed on isolated rerun,
and one unrelated lifecycle failure reproduced unchanged in the original
checkout.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- GitHub Actions scorecards now add an **E2E Test Phase Runtime**
section, including a semantic phase runtime table and slowest-phase
details.
- Runtime reporting now includes each phase’s **duration and outcome**,
with the slowest observed phase’s outcome shown as well.
- **Bug Fixes**
- If runtime progress artifacts are missing or invalid, scorecards
render a **fallback “runtime unavailable”** message instead of failing.
- **Documentation**
- Updated E2E documentation to reflect the expanded runtime tables and
enriched scorecard summary behavior for scheduled/manual runs.
- **Tests**
- Expanded E2E runtime and workflow boundary validations for the new
section, artifact handling, and outcome reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
cv added a commit that referenced this pull request Jul 23, 2026
<!-- markdownlint-disable MD041 -->
## Summary

#7100 established semantic phase coverage for all live E2E cases. This
follow-up closes the remaining contract gaps without reapplying that
implementation: it covers workflow-selected credential-free integration
tests, requires E2E child-process boundaries to expose content-free
liveness, and blocks OpenShell gateway-auth artifact uploads unless a
fail-closed safety scan approves the current run attempt. Together with
#7100, this completes #7101 while preserving current `main` behavior in
the audited overlap areas.

## Related Issue

Fixes #7101

## Changes

- Strengthen the shared progress contract with a frozen canonical
capability, target/scenario identity, total and phase elapsed time,
validated content-free events and activities, timestamp-only
child-output observations, and final-phase enforcement.
- Add `spawnObservedChild` as the audited direct asynchronous process
boundary used by E2E helpers. The semantic checker rejects unaudited
direct process APIs and requires synchronous calls to use a bounded
timeout plus `SIGKILL`.
- Add the lightweight `workflow-e2e-test` fixture for credential-free
integration tests selected by the authoritative E2E planner, so they
publish and validate the same semantic timeline and progress artifact
contract without the stateful live fixture.
- Convert the remaining agent-turn, Bedrock, Ollama, inference-routing,
runtime-override, fake-server, Docker, and cleanup process paths to
progress-aware boundaries without forwarding child stdout or stderr
contents.
- Scan final OpenShell gateway-auth artifacts before upload in both main
and candidate workflows. Unsafe files are quarantined or deleted, and
upload requires a run-ID/run-attempt-specific approval marker so a stale
or failed scan cannot authorize publication.
- Expand semantic coverage tests, workflow-boundary tests, pre-commit
routing, and contributor/E2E documentation for these additional
enforcement boundaries.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check one tests line and one docs line. Check other lines when
applicable. Add every requested justification or approval reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: An independent
clean-delta and rollback audit confirmed that #7100 behavior remains
intact, all 13 previously identified #7280-sensitive paths preserve
current `main`, the Hermes shard implementations are unchanged, and the
gateway-auth scanner/upload path fails closed. Focused scanner and
workflow-boundary tests pass on the rebased commit.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## DGX Station Hardware Evidence
<!-- Required only when scripts/prepare-dgx-station-host.sh changes.
Maintainers must review the linked evidence before approving or merging.
This is human-reviewed evidence, not authenticated hardware provenance.
Exceptional bypasses use existing repository governance and must be
documented on the PR. -->
- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — rebased final-head E2E-support suite:
17 files and 150/150 tests passed; `npm run test:e2e-phases:check`: 123
tests across 82 files passed; selected integration suite: 6 files passed
and 1 skipped, with 37 tests passed and 3 skipped; CLI and plugin
type-checks, CLI build, and diff checks passed
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — local `npm run check` completed the full
pre-commit stage, but its intentionally serial CLI/integration coverage
lane reached the four-hour execution ceiling before producing a final
summary; required sharded CI is authoritative
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — passed
with 0 errors and 2 existing Fern warnings
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added structured, redacted progress reporting across E2E phases and
workflow-selected credential-free integration tests.
* Introduced an OpenShell gateway auth artifact safety scan that
produces an approved payload and gates evidence/artifact uploads.

* **Bug Fixes**
* Prevented sensitive values from appearing in progress output and
persisted artifacts; uploads now proceed only after safety approval
succeeds.
* Hardened E2E subprocess lifecycle handling with forced termination,
bounded output capture, and safer shutdown behavior.

* **Documentation**
* Updated E2E phase-plan/progress and artifact-safety guidance,
including the `test:e2e-phases:check` contract.

* **Tests**
* Expanded coverage for progress reporting, observed subprocess
lifecycle, and workflow/semantic-phase boundary rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Documentation Writer Review
- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `CONTRIBUTING.md`, `test/e2e/README.md`,
`test/e2e/docs/README.md`
- Agent: Codex Desktop
- PR: #7397
<!-- docs-review-head-sha: bd75ae8 -->
<!-- docs-review-agents-blob-sha: 560ff38 -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
cv added a commit that referenced this pull request Jul 23, 2026
<!-- markdownlint-disable MD041 -->
## Summary

The PR E2E gate now creates one fresh child run when the first child
fails only because a standard GitHub-hosted runner disappeared. It
recognizes both GitHub's canonical lost-communication annotation and the
exact shutdown-log signature observed in the repeated Hermes
cancellations. Assertion, timeout, OOM, policy, mixed, custom-runner,
and incomplete-evidence failures remain terminal.

## Related Issue

Part of #7146. Parent epic: #7140.

## Changes

- Require a completed failed first child run, exact workflow identity,
complete Jobs and Checks pagination, and either the canonical SHA-bound
lost-runner annotation or one exact generic-cancellation annotation plus
authenticated terminal shutdown evidence.
- Authenticate the fallback through GitHub's job-log API without
forwarding the repository token to storage. Allowlist the signed HTTPS
result host, require a strong ETag and exact bounded range, and accept
only the terminal runner shutdown/cleanup block bound to the cancelled
step and job timestamps. Cap annotation count, per-field bytes, and
normalized aggregate evidence.
- Require every non-passing job to match the standard `ubuntu-latest`
GitHub Actions runner, runner group, check, annotation, and terminal
step-shape contract. Re-read and fingerprint all run, check, annotation,
and optional log evidence before dispatch.
- Create a fresh child run and replacement required check for the one
retry. Preserve the first run, check, state, plan, and evidence as
linked diagnostic history.
- Revalidate the PR SHA, base SHA, workflow SHA, risk plan, state, and
retry history before each privileged mutation.
- Keep coordination checks immutable across base identities so a
controller that resumes after a rapid retarget cannot fail the newer
base's gate.
- Terminalize the second attempt and clean partial replacement state
after failure, cancellation, skipped setup, interruption, or response
loss. The required observer follows the latest exact check.
- Serialize gate mutations by repository, PR number, PR SHA, and base
SHA. Keep up to 100 pending exact-identity mutations in GitHub's FIFO
queue without canceling their owner.
- Bound each child wait at 140 minutes, the controller at 330 minutes,
the observer at 358 minutes, and the required job at 360 minutes.
- Document the strict retry criteria, lineage, cleanup behavior, and
timeout envelope in `test/e2e/README.md`.
- Keep merged #7100 progress telemetry and #7399 resource telemetry
unchanged. This PR does not add a second sampler or retry final-main
runs.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: an independent
9-category audit passed at `697d78e5f`; it verified annotation and log
authentication, token isolation, bounded transport, spoof rejection,
immutable retry lineage, two-pass evidence revalidation, cross-base
check ownership, cleanup, timeout arithmetic, and FIFO mutation
ownership with no unresolved finding.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `test/e2e/README.md`
- Agent: Codex Desktop
- PR: #7423
<!-- docs-review-head-sha: d1f8247 -->
<!-- docs-review-agents-blob-sha: 560ff38 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 14 PR-gate integration files passed
244 tests at `697d78e5f`; the focused classifier/transport suites passed
72 tests; the test-only review fix passed all 26 controller tests at
`ac430dcf7`; the documented fallback lifecycle passed docs validation
and all 39 classifier tests at `d1f824728`; CLI and JavaScript type
checking, changed-file hooks, source-shape, test-size, Biome, YAML,
Markdown, gitleaks, and diff checks passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — targeted controller coverage is
authoritative; GitHub CI will run the repository gate.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — passed
with 0 errors and 2 existing Fern warnings
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only) — no `docs/` page changed
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— no new doc page

---

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a one-time automated PR E2E retry for hosted-runner loss,
including dedicated retry and abandon lifecycle steps with evidence
download/verification.
* Updated PR E2E gate coordination to run retry-sensitive paths only on
the first workflow attempt using attempt-aware, per-PR/per-SHA
concurrency isolation.
* **Bug Fixes**
* Hardened gate/approval execution to prevent cross-run collisions and
incorrect retry/authorization, with improved fail-closed behavior.
* Increased relevant time budgets and tightened “superseded/cancel”
handling.
* **Documentation**
* Expanded PR E2E gate flow documentation for retries, evidence rules,
and terminal outcomes.
* **Tests**
* Expanded end-to-end coverage for runner-loss classification, retry
lifecycle, authorization, and command parsing edge cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
cv pushed a commit that referenced this pull request Jul 23, 2026
## Summary

Records bounded start/end CPU, memory, and workspace evidence for every
#7145 larger-runner route so the standard-versus-larger experiment can
compare like-for-like jobs. This is the current-`main` replacement for
the unique telemetry portion of closed #7247; it does not change runner
routing or duplicate #7100 phase, stall, timing, or resource
diagnostics.

## Related Issue

Related to #7145. Supersedes the telemetry portion of #7247.

## Changes

- Write one canonical private numeric sample after workspace preparation
and one from an `always()` finalizer before artifact scanning or upload.
- Cover 12 routed workflow lane identities and 13 concrete executions;
`hermes-inference-switch` contributes its `hosted` and `anthropic`
matrix entries.
- Keep the OpenClaw entries for `mcp-bridge`, `channels-stop-start`, and
`security-posture` uninstrumented because #7391 leaves those entries on
standard runners.
- Summarize measured-window CPU utilization, host-memory endpoints and
root-cgroup lifetime peak, plus workspace endpoints and growth without
invoking Docker, `ps`, or background samplers.
- Reject unknown fields, duplicate JSON keys, malformed timestamps,
identity/counter drift, oversized evidence, symlink/hardlink final-file
substitution, matrix topology drift, and workflow scope/order drift.
- Document comparison controls and limitations. Standard-runner
baselines can begin before #7391 changes routing; the same evidence
remains after larger-runner activation.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent review
confirmed the exact trusted-main/no-alternate-checkout boundary,
Hermes-only matrix guards, bounded private schema, and
finalizer-before-scan/upload ordering.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed
implementation
- Result: `docs-updated`
- Evidence: `test/e2e/README.md` documents the 12 routed workflow lane
identities, 13 concrete executions, OpenClaw exclusions, sample
limitations, missing-summary interpretation, and comparison
requirements. The exact-head refresh confirmed the final test-only
review fix requires no further documentation change.
- Agent: Codex Desktop
- PR: #7399
<!-- docs-review-head-sha: c09137d -->
<!-- docs-review-agents-blob-sha: 9061453 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — focused E2E-support validation passed
102 tests; an independent rerun of the four directly affected suites
passed 94 tests; the final matrix-assertion review fix passed its
focused 28-test suite.
- [ ] Applicable broad gate passed — Full `e2e-support` previously
reached 1,381 passed and 9 skipped; two unrelated existing local
failures remained. GitHub CI is authoritative for the Linux gate.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 existing Fern warnings
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only) — no `docs/` page changed
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— no new doc page

---

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
apurvvkumaria added a commit that referenced this pull request Jul 24, 2026
<!-- markdownlint-disable MD041 -->
## Summary
This adds bounded periodic and phase-aware resource telemetry to the
canonical runner-comparison ledger for trusted heavy E2E executions. It
reuses the existing progress pulse and records only numeric or
fixed-enum evidence, improving runner-pressure diagnosis without
changing failure classification or retry policy.

## Related Issue
Related to #7145 and #7146 (parent epic: #7140).

## Changes
- Extend the canonical runner-comparison artifact to a bounded v2
lifecycle with private `0600` files, a reserved final sample, v1 read
compatibility, and deterministic summaries.
- Capture bounded host, cgroup, PSI, filesystem, Docker, container, and
coarse process-class evidence at endpoints, phase boundaries, and an
approximately 60-second cadence.
- Reuse the existing progress timer with fixed deadlines, no catch-up
bursts, per-scenario attribution boundaries, and a permanent fallback to
the existing bounded collector if canonical sampling fails.
- Add focused privacy, compatibility, timing, fallback, aggregation,
summary, and workflow-boundary coverage, including the three-shard
`common-egress-agent` topology merged in #7448.
- Document the telemetry lifecycle, probe budgets, privacy limits,
compatibility behavior, and the separation from classification and
retry. This builds on the merged progress/timing work from #7100 and
#7213 and extends the canonical ledger from #7399 rather than creating
parallel infrastructure.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check one tests line and one docs line. Check other lines when
applicable. Add every requested justification or approval reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent exact-head
privacy, schema, classifier, retry, and compatibility review at
`419182d89` found no actionable issues.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review
<!-- Required for code and documentation changes after the changes and
applicable validation are complete. Keep one review checkbox and one
instance of each visible or hidden field. For Evidence, list changed
documentation paths. For documentation-only changes, also state that the
writing rules and documentation style were reviewed. For other results,
explain why no documentation change is needed or why the review is
blocked. For Agent, use a consistent product and surface name, such as
Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all
review changes, put `git rev-parse --short HEAD` and `git rev-parse
--short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review
and refresh that metadata after any new commit. This receipt is advisory
during the data-collection pilot. -->
- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `test/e2e/README.md`
- Agent: Codex Desktop
<!-- docs-review-head-sha: e06653c -->
<!-- docs-review-agents-blob-sha:
9c9b36d -->

## DGX Station Hardware Evidence
<!-- Required only when scripts/prepare-dgx-station-host.sh changes.
Maintainers must review the linked evidence before approving or merging.
This is human-reviewed evidence, not authenticated hardware provenance.
Exceptional bypasses use existing repository governance and must be
documented on the PR. -->
- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
Exact-head serialized telemetry/progress/workflow suite passed 225/225
tests across 15 files; after the linear-test guardrail repair, the
primary focused suite passed 165/165 across 9 files, Vitest membership
was exact across 1,764 files, CLI typecheck passed, and `npm run
check:diff` passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: A broad E2E-support run
exposed only parallel-load timeouts and one macOS lifecycle failure
reproduced unchanged on `origin/main`; scoped exact-head gates are
listed above.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added runner-comparison schema v2 telemetry as a bounded, ordered
ledger with periodic, scenario-start, phase, and finalize samples.
* Enabled live E2E runner-comparison sampling and added conditional
periodic sampling in the E2E progress harness.
* Introduced pulse-based progress resource sampling with configurable
interval and best-effort phase/scenario capture.

* **Bug Fixes / Improvements**
* Switched memory diagnostics to available-memory reporting (with
free-memory fallback labeling).
* Hardened telemetry parsing/validation, evidence sanitization, and
expanded runner-pressure/resource diagnostics.

* **Documentation**
* Documented v2 ordering/cadence, privacy/size limits, and
diagnostic-only summary guidance.

* **Tests**
* Added/expanded E2E tests for v2 schema validation, timing/cadence,
fallback behavior, and updated expected outputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure feature PR adds or expands user-visible functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants